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ABSTRACT 
Differential  game  theory  and  dynamic  programming  are  applied 
to  derive  the  optimal  strategy,  or  sequence  of  controls,  for  a  class 
of  linear,  sampled-data ,  pursuit-evasion  problems.     The  necessary 
and  sufficient  conditions  for  existence  of  the  solution  are  derived. 
A  digital  computer  program  for  simulating  control  generation  and  sys- 
tem trajectories  is  given.     The  results  of  simulation  tests  using  this 
digital  computer  model  are  presented  to  demonstrate  the  salient 
characteristics  of  the  strategy. 
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1 .    INTRODUCTION 

Recent  literature  in  the  field  of  automatic  control  theory  shows  an 
increasing  interest  in  the  application  of  differential  game  theory  to 
control  system  design  [  1,  2,   3  ]  .     This  approach  to  control  system 
design  may  be  considered  an  extension  of  "optimal"  control  theory 
allowing  one  or  more  opposing  systems  controlled  by  intelligent 
opponents  to  be  introduced  into  the  design  computations.     ("In- 
telligent" is  used  here  to  mean  having  knowledge  of  the  opponent's 
system  dynamics,  states,  and  desires.)    Further,  this  approach  assumes 
that  the  controller  of  each  system  will  act  in  accordance  with  his  in- 
telligence.   This  allows  system  design  to  provide  satisfactory  per- 
formance when  an  opponent  acts  to  minimize  that  performance,  and  to 
provide  improved  performance  when  the  opponent  acts  otherwise. 

The  disadvantages  of  this  approach  to  control  system  design  are 

generally  those  attributed  to  optimal  control  theory  in  the  past 

the  difficulty  of  expressing  realistic  performance  objectives  in  the 
mathematical  form  of  a  performance  index,  and  the  difficulty  of  ob- 
taining solutions  to  problems  with  realistic  physical  constraints. 
These  difficulties  are  somewhat  increased  by  the  introduction  of  in- 
telligent opponents. 


2  .     STATEMENT  OF  THE  PROBLEM 

This  research  was  confined  to  exploring  the  solution  and  salient 
characteristics  of  the  sampled-data  analog  of  a  continuous  system 
pursuit-evasion  problem  and  solution  published  by  Y.C.  Ho,  A.E. 
Bryson,  and  S.  Baron  [  3  ]  . 
2.1    Differential  Constraints 

The  pursuit  and  evasion  plants  are  described  by  a  set  of  linear, 
time-invariant,  first-order  state  equations  [4  ]: 

x      =    A  x      +    B   u  (1) 

-p  p-p  p- 

x      =    A   x      +    B   v  (2) 

— e  e--e  e— 

The  following  definitions  apply  to  equations  (1)  and  (2): 

x       is  the  (n  by  1)  state  vector  of  the  pursuit  plant 

x        is  the  (n  by  1)  state  vector  of  the  evasion  plant 

u        is  the  (m     by  1)  control  vector  of  the  pursuit  plant 
_  P 

is  the  (m     by  1)  control  vector  of  the  evasion  plant 

A     and  A     are  (n  by  n)  matrices  of  constant  coefficients 
P  e 

B       is  an  (n  by  m   )  matrix  of  constant  coefficients 
P  P 

B        is  an  (n  by  m   )  matrix  of  constant  coefficients. 
e  e 

Since  the  controls  u  and  v  are  to  be  applied  in  sampled-data  form, 

it  is  convenient  to  describe  the  plants  by  the  following  difference 

constraint  equations  [4  ]: 

x    (k+1)    =    0   (T)x    (lc)    +    A    (T)u(k)  (3) 

-p  P      -p  P 

x    (k+1)    =    0    (T)x    (k)    +    A    (T)v(lc)  (4) 

— e  e      — e  e      — 

where  0    and  0    are  state  transition  matrices  defined  by: 
p  e 

0   (T)    =    eAPT  (5) 

P 

0   (T)    =    eAeT       where,  (6) 

2    2  3    3 

AT  AT         A   T  A_JT_  m 

6  =    l    +    IT    +    ~7T  +   TV  +    "    '    '  (7) 


The  state  distribution  matrices  are  defined: 


A    (T)  =    JT  0   (T-t)B  dt 
P  0        P  P 

A    (T)  =    J.T0   (T-t)B  dt      . 
e  ^0       e  e 


(8) 
(9) 


The  integrals  (8)  and  (9)  can  be  evaluated  by  the  infinite  series: 


A(T)    = 


AT2         A2T3         A3T4 
IT    +  -7T7-    +      „.      +    -^rr-  +  .    .    . 


2! 


3! 


4! 


(10) 


2.2   Performance  Index 

The  performance  index  is  defined: 


J  =  cxpty-x^V3  'qlvV-W  ] 


+    J0  f  CuT(t)R    (t)u(t)  -  vT(t)Rev(t)  ]dt        . 


(11) 


The  following  additional  definitions  apply  to  the  performance  index: 
Q     is  a  (n  by  n)  positive  semi-definite,  symmetric,  final- 
state-difference  weighting  matrix. 

R      is  a  (m     by  m   )  positive  definite,  symmetric,  pursuit 
P  P  P 

control  weighting  matrix. 

R      is  a  (m     by  m   )  positive  definite,  symmetric,  evasion 
e  e  e 

control  weighting  matrix. 
Since  the  controls  are  to  be  applied  in  sampled-data  form 
(u  and  v  constant  for  the  sample  period  T)  for  N  sample  periods 
(t,  =  NT),  the  performance  index  given  in  (11)  is  represented  by: 


J    =    [x    (N)  -  x    (N)  ]      Q[x    (N)  -  x    (N)   ] 
— p            — e  — p  — e 

N-I 

+  T    L  [u    (k)R   (k)u(k)    -    v    (k)R    (k)v(k)  ]     .  (12) 

k=0       "  P      "  -  e      - 


The  desired  optimal  solution  is  a  set  of  controls: 


and 


u      =    {u(0)   ,  u(l)   ,    .    .    .  u(N-l)   }     to  minimize  J, 


v      =    (v(0)   ,  v(l)   ,    .    .    .  v(N-l)  }     to  maximize  J. 


Equivalently ,  the  saddle  point 


J(u°  ,  v)    £  J(u°  ,  v°)    *  J(u,  v°) 


is  sought 


(13) 


It  is  assumed  at  this  point  that  such  a  saddle  point  does  exist 
and  that  the  optimal  value  of  the  performance  index  is: 

min 


t°/        »         t/  °     °\         min 
J    (u,v)    =    J(u    ,v   )    = 


max 
v 


J 


max 
v 


u 


J 


(14) 


The  dominant  characteristics  of  this  performance  index  are  as 
follows: 

1)  The  form  of  the  performance  index  requires  that  the  orders  of 

the  pursuit  and  the  evasion  plants  be  the  same  (otherwise  x    and  x 

— p  — e 

would  not  be  conformable  for  subtraction).    The  dimensions  of  the 
pursuit  and  evasion  control  vectors,  however,  need  not  be  the  same 

(m     need  not  equal  m   ) . 

P  e 

2)  The  final  state  difference  weighting  matrix  Q  is  chosen 
positive  semi-definite  in  order  to  reflect  an  increase  in  the  performance 
index  for  an  increase  in  the  difference  of  the  states  of  interest  only 

of  the  two  plants  at  the  final  time. 

3)  The  R    matrix  is  chosen  positive  definite  in  order  to  reflect 

P 

an  increase  in  the  performance  index  for  the  use  of  any  pursuit  control. 

4)  The  R    matrix  is  chosen  positive  definite  in  order  to  reflect 

e 

a  decrease  in  the  performance  index  for  the  use  of  any  evasion  control. 

5)  The  matrices  0,  R    ,  and  R    are  chosen  symmetric  since  this 

p  e 

choice  involves  no  loss  of  generality  in  the  quadratic  form  but  makes 
subsequent  statements  about  the  solution  simpler. 
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3 .     DERIVATION  OF  THE  OPTIMAL  STRATEGY 
The  derivation  given  here  was  first  done  by  Dr.  Donald  E.  Kirk, 
Assistant  Professor  of  Electrical  Engineering  at  the  Naval  Postgraduate 
School.    It  has  been  expanded  slightly  to  include  a  set  of  sufficiency 
conditions  and  to  make  its  application  as  general  as  possible. 
3.1   Minimaximization  of  the  Performance  Index 

The  optimal  controls  are  found  by  applying  the  dynamic  pro- 
gramming approach  of  Bellman  [  5  ]  .     The  "cost"  of  operation  during 
the  final  (Nth)  stage  of  operation  is  given  by 

J.     =    [x    (N)  -  x    (N)   ]T  Q[x    (N)  -  x    (N)   ] 
1  h?  — e  -p  — e 

+  uT(N-l)R    (N-l)u(N-l)  -  vT(N-l)R   (N-l)v(N-l)   .  (15) 

-  p  -  -  e  - 

The  effect  of  the  sample  period  in  equation  (12)  can  be  accounted 

for  by  multiplying  each  element  in  the  R    and  R    matrices  by  T.    T  is 

p  e 

omitted  for  convenience  at  this  point  to  be  reintroduced  later  in  the 

derivation.     Since  x    (N)  and  x   (N)  depend  only  on  x   (N-l),  x   (N-l), 

— p  --e  — p  — e 

u(N-l),  and  y(N-l),  substituting  equations  (3)  and  (4)  into  equation 
(15)  and  neglecting  the  indices,  which  are  all  (N-l),  yields 

h     =    £^xr,    +    Ar,u    "    t    "    A   v^TQ^x^+A^-0x     -Av] 
1  p— p  p—  e-^e  e—  p— p        p—        e— e        e— 

+    uTR  u    -    vTR  v       .  (16) 

-     p-        -     e- 

It  is  convenient  to  define: 

z(N-l)    =    0  x    (N-l)    -    0  x    (N-l)   .  (17) 

—  p— p  e— e 

Note  that  the  _z(N-l)  vector  is  the  projected  difference  in  states  at 

the  final  time  if  no  control  is  applied  during  the  final  (Nth)  stage  of 

operation. 

Using  equation  (17)  in  (16)  and  defining  P(0)    =    Q: 

J,     =    [z+Au-Av  ]TP(0)[  z  +  A   u  -  A   v  ] 
1  —         p—         e—  —         p—         e— 

T  T 

+  uRu-vRv      .  (18) 

-     p-     -     e- 

11 


Defining  dj   (du)  and  dj    (dv)  as  the  increments  of  J     from  its  optimal 
value  caused  by  small  variations  du(N-l)  and  dv(N-l),  respectively, 
from  the  optimal  values  u    (N-l)  and  v    (N-l)  gives: 

dj.  (du)  =  2  {[A  TP(0)A    +R    ]u°  -  A  TP(0)A  v°  +  A  TP(0)z}  Tdu 
1-  p  pp-p  e-p-- 


+  2duT[A  TP(0)A    +  R   ]du 
-        p  p        p      - 


(19a) 


dj.  (dv)  =  2{  [A     P(0)A    -  R   ]v°  -  A  TP(0)A  u°  -  A  TP(0)z}    dv 
1     -  e  e        e  —  e  p—  e  —        — 


+  2dvT[A  TP(0)A    -  R    ]dv 
—       e  e        e      — 


(19b) 


Inspection  shows  that  the  necessary  conditions  for  the  desired 
saddle  point  are  that  the  first  order  terms  in  du  and  dv  in  equations 
(19)  must  be  zero.    Then: 


[A  TP(0)A    +  R    ]u°  -  A  TP(0)A  v°  +  A  TP(0)z    =    0 
pv/pp-  p  e-  p  -- 


(2  0) 


-A  TP(0)A  u°  +  [A  TP(0)A    -  R    ]v°  -  A  TP(0)z    =    0 
ep—  e  e        e   —  e  —        — 


(21) 


suming  that  the  inverse  required  below  exists  ,  and  solving  for 

-1 


o        ,     o 
u     and  v     gives 


o 


A     P(0)A    +  R 
P  P        P 


-A     P(0)A 
e  p 


-A     P(0)A 
P  e 


A     P(0)A    -  R 
e  e        e 


-A  TP(0) 

___P 

A  TP(0) 
e 


t  is  convenient  to  rewrite  equation  (22)  in  the  form: 

z(N-l) 


u°(N-l) 

- 

F    (1) 
_  P 

v°(N-l) 

FAX) 

e 

(22) 


(23) 
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Substitution  of  the  above  expression  for  the  optimal  controls  u  (N-l) 
and  v  (N-l)  into  equation  (18)  yields  the  optimal  "cost"  of  operation 
over  the  last  (Nth)  stage  of  operation: 

J1°[z(N-l)]=  [zT  +  (A  F  z  -  A  F  z)T]  P(0)[z  +AFz-AFz] 
1     —  —  p  p—        e   e—  —        p  p—        e  e— 


T     T  T      T 

+  zF     RFz-zF     Rz 
—     p     p  p—      —     e     e— 


Now  defining    ib  (1)  =  I  +  A  F    (1)  -  A  F    (1) , 
r  p  p  e   e 

equation  (24)  can  be  rewritten: 

T  °[z(N-l)]=zTi/)T(l)P(O)0(l)z  +  zTF  T(1)R    (N-l)F    (l)z 
1—  —  —      —     pp  p      — 


(24) 
(25) 


-  zTFoT(l)R   (N-l)F    (l)z 
—     e  e  e      — 


(2  6) 


Now  defining 

P(i)  =  0T(i)P(i-l)iMi)  +  F  T(i)R    (N-i)F    (i)  -  FoT(i)R    (N-i)F    (i)   , 

p  p  p  e  e  e 


(27) 


equation  (2  6)  can  be  rewritten: 


J^CzjN-l)]   =    zT(N-l)P(l)z(N-l)   . 


(28) 


The  dynamic  programming  process  is  continued  by  considering 
the  cost  of  operation  over  the  last  two  stages  of  operation. 

J0  =  uT(N-2)R    (N-2)  u(N-2)  -  vT(N-2)R    (N-2)v(N-2)  +  J,  [z(N-l)]       (29) 
2      —  p—  —  e—  1  — 


o      min 


u(N-l),  u(N-2) 


max 


v(N-l),  v(N-2)      J2 


(30) 


Since  u(N-l)  and  v(N-l)  affect  only  the  J     portion  of  equation  (29) 


o      mm 


u(N-2) 


max 
v(N-2) 

min 


uT(N-2)R    (N-2)u(N-2)  -  vT(N-2)R    (N-2)v(N-2) 
-  p  -  -  e  - 


u(N-l) 


max 


v(N-l)    Jl 
13 


(31) 


Equation  (31)  can  be  rewritten  using  the  definition  of  J       in 
equation  (14): 


mm 
u(N-2) 


v(N-2)      ^T(N-2)Rp(N-2)u(N-2) 


-  vT(N-2)R    (N-2)v(N-2)  +  J  °  ] 
—  e  —  1 


(32) 


Substituting  equation  (28)  into  (32)  gives: 


min 
u(N-2) 


v(N-2)   £uT(N-2)R    (N-2)u(N-2)  -  y_T(N-2)Re(N-2)v(N-2) 


+     z    (N-DP(l)z(N-l)] 


(33) 


Relating  z(N-l)  to  x    (N-2) ,  x    (N-2)   ,  u(N-2),  and  v(N-2)  yields: 
-p  -e  -  - 


z(N-l)    =    0  x    (N-l)  -  0  x    (N-l) 
—  p— p  e— e 


(34) 


z(N-l)  =    0    10  x(N-2)  +  A  u(N-2)]-  0    10  x    (N-2)  +  A  v(N-2)]       (35) 

—  p     p—  p—  e     e— e  e— 

z(N-l)  =  0  2x    (N-2)  -  0  2(N-2)  +  0  A  u(N-2)  -  0  A  v(N-2)     .  (36) 

—  p  — p  e  p  p—  e   e— 


Extending  the  definition  of  z ,  define: 

z(N-i)    =    0  lx    (N-i)  -  0  Sc    (N-i)     . 
—  p  — p  e  — e 


(37) 


It  is  noted  that  z_(N-i)  is  the  projected  "miss  distance"  if  no  control 

is  applied  to  either  the  pursuer  or  evader  after  the  ((M-i)th  stage  of 

operation. 

It  is  also  convenient  to  define: 

A  (i)    =    0  XA 
P  P     P 

A   (i)    =    0jL        . 
e  e     e 


(38) 
(39) 
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Substituting  equations  (37),   (38),  and  (39)  into  equation  (36)  gives: 


z(N-l)    =    z(N-2)  +  A   (l)u(N-2)  -  A  (l)v(N-2)    . 
-  -  p      -  e      - 

Substituting  equation  (40)  into  equation  (33),  and  omitting  all  (N-2) 

subscripts  gives; 


(40) 


J2    [*<N-2>]  =    u(N.2) 


max 
v(N-2) 


T  T 

u  R   u  -  v   R  v 
-     p-     -     e- 


+  [z  +  A  (l)u  -  A  (l)v]    P(l)[z  +  A  (l)u  -  A  (l)v] 
—        p—        e—  —        p—        e      — 


.   (41) 


Comparison  of  equation  (41)  and  equation  (18)  shows  them  to  be 
identical  in. form.  It  is  apparent  that  the  solution  shown  in  equation 
(23)  for  the  last  stage  of  operation  can  be  generalized  for  all  stages 
of  operation: 


u°(N-i) 
v°(N-i) 


Fe(i» 


z(N-i)    =    F(i)z(N-i)     . 


(42) 


The  recursive  relationships  necessary  to  compute  the  F(i)  matrices 
for  all  sample  periods  are  as  follows: 

P(0)       =     Q  (43a) 


A  (0)  =     A 
P  P 

A  (0)  =     Ao 
e  e 


(43b) 
(43c) 


Fp(i) 

rw 


A  T(i-l)P(i-l)A  (i-1)  +  R   (N-i)  i 
•  P  P  P  ! 


-A     (i-l)P(i-DA  (i-1) 
p  e 


-A     (i-l)P(i-l)A  (i-1) 
e  P 


A     (i-l)P(i-l)A  (i-1)  -  R   (N-i) 


-1 


x 


-a  "(i-DPdfi) 

_P 

AeT(i-l)P(i-l) 


(43d) 
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t  (i)    =    [I  +  A  (i-l)F    (i)  -  A  (i-l)F    (i)] 
p  p  e  e 


(43e) 


P(i)    =    </)T(i)P(i-l)j/)(i)  +  F  T(i)R    (N-i)F    (i) 

P  P  P 


-  F      (i)R   (N-i)F    (1) 
e  e  e 


A  (i)    =    0  A  (i-1) 
P  P  P 


A  (i)    =   0  A  (i-1) 
e  e   e 


(43f) 
(43g) 
(43h) 


The  relationships  necessary  to  generate  the  optimal  controls  are: 

z(N-i)    =    0  Xx    (N-i)  -  0  lx   (N-i)  (44a) 

—  p  - p  e  —  e 


u°(N-i) 
v°(N-i) 


IS. 

Fe(i) 


z(N-i) 


(44b) 


The  effect  of  the  sample  period  on  the  solution,  which  was 
neglected  after  equation  (12),  can  now  be  readily  accounted  for  by 

multiplying  each  element  of  the  R    and  R    matrices  by  the  sample 

P  e 

period. 

3.2    Sufficiency  Conditions 

It  has  been  assumed  thus  far  that  the  saddle  point  described  in 
equations  (13)  and  (14)  exists.     Examination  of  equations  (19)  shows 
that  such  a  saddle  point  is  indeed  reached  by  applying  the  control 
law  described  by  equations  (43)  and  (44)  if  the  following  conditions 
are  met: 

a)  the  first  order  terms  in  du  and  dv  in  equations  (19)  must  be 
identically  zero, 

b)  the  second  order  term  in  du  in  equation  (19a)  must  be 
positive  definite,  and 

c)  the  second  order  term  in  dv  in  equation  (19b)  must  be 
negative  definite . 
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It  is  convenient  here  to  identify  the  matrix  which  requires  inversion  in 
equation  (43d)  as  the  matrix  D,  and  the  sub-matrices  into  which  it  is 
partitioned  as  follows: 


D    = 


D 


11 


D 


21 


D 


12 


D 


22 


(45) 


An  equivalent  set  of  sufficient  conditions  can  now  be  stated  as 
follows: 

a)  the  inverse  of  the  D  matrix  must  exist, 

b)  the  matrix  D       must  be  positive  definite,  and 

c)  the  matrix  D       must  be  negative  definite. 

A  moment's  examination  of  equation  (43d)  reveals  that  the  D 
matrix  is  symmetric.    It  can  be  shown  that  the  inverse  of  the  symmetric 
matrix  D  exists  if  the  inverse  of  D       and  the  inverse  of    4  exist  [  6  ]  , 
where: 


i=    D22-D21(V"lD12     ' 


(46) 


Therefore,  the  necessary  and  sufficient  conditions  to  insure  the 

required  saddle  point  are: 

T 

a)  A     (i-l)P(i-l)A  (i-1)  +  R   (N-i)  must  be  positive  definite 

P  P  P 

for  all  i  from  one  to  N,  and 

T 

b)  A     (i-l)P(i-DA  (i-1)  -  R   (N-i)  must  be  negative  definite 

e  e  e 

for  all  i  from  one  to  N. 
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4.     DIGITAL  COMPUTER  SIMULATION  PROGRAM 
In  order  to  simulate  the  results  of  applying  the  control  strategy 
derived  in  this  thesis  to  various  systems,  the  digital  computer  pro- 
gram OPTIMAL2  was  employed.     This  program  was  written  in  FORTRAN 
63  computer  language  for  use  with  a  Control  Data  Corporation  1604 
Computer.     The  complete  program  is  given  in  Appendix  A. 

The  program  OPTIMAL2  was  written  to  have  as  general  an  appli- 
cation as  possible  within  the  restrictions  imposed  in  the  problem 
statement  and  derivation.    To  this  end,  pursuit  and  evasion  system 
descriptions  and  the  performance  index  to  be  minimized  are  read  in  as 
data.     (The  pursuit  and  evasion  systems  are  completely  described  by 

the  differential  equation  coefficient  matrices  A    ,  A    ,   B    ,   B    ,  and  the 

p       e       p       e 

sample  period  T.     The  performance  index  to  be  minimized  is  complete- 
ly described  by  the  weighting  matrices  Q,  R    ,  R    ,  the  sample  period 

P       ® 
T,  and  the  number  of  samples  N.     The  initial  conditions  x    (0)  and 

~P 

x    (0)  are  required  for  trajectory  simulation  only.)    The  program  will 
— e 

accept  any  system  so  described  with  state  vector  dimensions  of  ten 

or  less  and  with  control  vectors  of  dimensions  such  that  the  sum  for 

both  plants  does  not  exceed  ten.     Printout  of  input  data  is  provided 

for  verification  purposes. 

The  program  uses  subroutine  PHIDEL  to  compute  the  matrices  0   , 

0    ,  A     ,  and  A    for  the  systems  described  by  the  input  data.    This 
e       p  e 

computation  is  effected  by  programming  truncated  forms  of  the  in- 
finite series  given  in  equations  (7)  and  (10).     Printout  of  these 
matrices  is  provided  for  reference  purposes. 

The  program  uses  subroutine  OPTCON2  to  generate  and  record 
on  magnetic  tape  for  later  use  the  "feedback  gain"  matrices  F    (i) 

and  F    (i)  and  the  state  transition  matrices  0      and  0     .    These 
e  p  e 

comoutations  are  effected  by  programming  equations  (43).     Comment 

statements  in  the  subroutine  indicate  the  appropriate  location  for 

additional  statements  required  to  read  from  data  or  to  generate  a 

"time-varying"  form  of  the  control  weighting  matrices  R   (N-i)  and 

P 
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R    (N-i) .     (No  correction  is  required  if  these  matrices  are  constant.) 
e 

Provision  is  made  for  printout  of  a  number  of  the  feedback-gain  matrices 
and  the  state  transition  matrices  ,  and  for  graphical  output  of  the  (1 , 1) 
elements  of  the  feedback-gain  matrices  to  show  trends  in  their  variation 
Provision  is  also  made  for  a  printout  indication  if  the  sufficiency  con- 
ditions for  the  required  saddle  point  are  not  met. 

The  program  then  applies  the  matrices  generated  by  OPTCON2  to 
compute  the  sequence  of  optimal  controls  by  programming  equations 
(44),  and  generates  a  simulated  trajectory  for  each  system.    The 
value  of  the  performance  index  is  computed  for  these  trajectories, 
and  is  provided  as  a  printed  output.    Comment  statements  near  the 
end  of  the  program  OPTIMAL2  indicate  where  changes  of  the  output 
statements  may  be  required  to  present  the  trajectories  of  different 
systems  in  usable  form. 

Figure  1  shows  a  vector  flow-graph  model  for  implementing  the 
optimal  pursuit-evasion  strategy.    The  optimal  strategy  is  defined 
(the  feedback-gain  matrices  may  be  computed  for  the  largest  number 
of  sample  periods  likely  to  be  required)  when  the  dynamics  of  the 
pursuit  and  evasion  plants  are  known,  and  it  may  be  recorded.     There- 
fore,  "on-line"  or  "real-time"  computation  of  the  strategy  is  not 
required. 

Examples  of  the  required  form  of  input  data  and  examples  of 
printed  output  are  presented  in  Appendix  B.     Examples  of  variations 
on  the  basic  program  for  different  systems  are  presented  in  Appendix 
C. 
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PURSUER 


up(k) 


F    (N-k) 
P 


F    (N-k) 
e 


u    (k) 
— e 


x    (k) 
- e 


EVADER 


VECTOR  FLOWGRAPH  MODEL 
FIGURE  1 
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5.     RESULTS  OF  COMPUTER  SIMULATION  TESTS 
Using  the  computer  simulation  program  described  in  the  pre- 
ceding section,  three  series  of  tests  were  run  to  investigate  different 
characteristics  of  the  optimal  pursuit-evasion  strategy.     The  results  are 
reported  below. 

The  following  definitions  are  used  to  simplify  the  recording  of 
data: 

(47) 


z(N)    =    x   (N)  -  x   (N)   , 
-  -p  -e 

N 

Cost  R    =     I     [u   (k)R  u(k)  -  v   (k)R  v(k)]  , 

k=0      "  P~ 


Cost  Q    =    z    (N)Qz(N)   ,  and 


(48) 
(49) 


J    =    Cost  R  +  Cost  Q. 


(50) 


5 . 1    The  Effect  of  Varying  the  Performance  Index  Parameters 

Run  1  is  a  standard  intercept  to  which  subsequent  runs  are  gen- 
erally compared.    The  pursuit  and  evasion  plants  are  identical,  un- 
damped, spherical,  unit  masses  traveling  in  a  two-dimensional 
(north-east)  space.     The  state  vectors  and  control  vectors  for  these 
plants  are  defined: 


x  = 


north 
north 
east 
ea'st 


and 


u  = 


north 
east 


(51) 


The  computer  input  data  and  terminal  performance  data  are  given 
below.    The  resulting  trajectories  and  the  time  variation  of  the  (1 , 1) 
elements  of  the  feedback-gain  matrices  are  shown  in  Figures  2  and  3. 
A  complete  set  of  input  data  showing  the  required  format  and  a  com- 
plete computer  printout  of  the  numerical  results  for  this  run  are  given 
in  Appendix  B. 
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Input  Data 


A   =A   = 
P      e 


0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

B   =B   = 
P     e 


0 

0 

1 

0 

0 

01 

0 

1 

Q.= 


1 

0 

0 

o" 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

R     = 


1    0 
0    1 


R     = 
e 


5    0 
0    5 


V 

x    (0)  = 

5 
0 
5 

x    (0)  = 
— e 


500 

-10 

0 

10 


N    =    20  , 

T     =       1. 


Terminal  Performance  Data 
.0938" 


z(N)= 


30.0020 

.0469 

2.5012 


Cost R     =      23.430 


Cost  Q     = 
J      = 


.011 
23.441 


The  use  of  this  particular  semi-definite  Q  matrix  defines  position 
only  as  tiie  "states-of-interest"  for  the  performance  index.     The  effect 
of  this  choice  is  readily  apparent  in  the  resulting  relative  difference 
in  the  magnitudes  of  the  velocity  and  position  elements  of  the  state 
difference  vector  at  the  final  time,  z_(N) ,  and  in  the  "intercept"  tra- 
jectories in  Figure  2. 

Examination  of  the  series  of  feedback-gain  matrices  (see 
Appendix  B) ,  which  define  the  optimal  strategy  for  this  run,   shows 
that  the  required  acceleration  for  the  pursuer  is  always  parallel  to  the 
projected  terminal  "miss-distance"  vector,  so  as  to  provide  maximum 
reduction  in  the  projected  miss-distance  for  a  given  acceleration 
magnitude.    The  required  acceleration  for  the  evader  is  ih  the. same 
ection  (and  smaller  in  magnitude).    This  situation  is  peculiar  to 
the  special  case  where  both  systems  are  point  masses,  the  controls 
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Figure  2 
Standard  Intercept  Trajectories 
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Figure  3 

Time  Variation  of  the  (1,1)  Elements 

of  the 

Feedback-gain  Matrices 

for  the 

Standard  Intercept 
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are  accelerations,  and  the  final  state  weighting  matrix  Q  indicates 
interest  only  in  position  error  at  the  final  time.     This  physical  in- 
terpretation of  the  strategy,  previously  stated  only  in  mathematical 
form,  lends  an  intuitive  appeal. 
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Run  2  shows  the  effect  of  increasing  the  time  for  pursuit.  The 
input  data  and  the  terminal  performance  data  are  given  below.  The 
resulting  trajectories  are  shown  in  Figure  4. 

Input  Data 

Identical  to  Run  1  except:      N    =    25 . 

Terminal  Performance  Data 


z(N)  = 


.030 

22.501 

■      .030 

2.500 


Cost  R     =    7.4994 
Cost Q     =       .0018 


J      =    7.5012 

The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 
that: 

a)  the  terminal  "miss-distance"  is  less,  and 

b)  smaller  magnitudes  of  acceleration  are  required. 

The  series  of  feedback-gain  matrices  is  identical  to  that  for  Run  1  . 
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Figure  4 

Intercept  Trajectories 

for 
Increased  Final  Time 
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Run  3  shows  the  effect  of  decreasing  the  time  for  pursuit.  The 
input  data  and  the  terminal  performance  data  are  given  below.  The 
resulting  trajectories  are  shown  in  Figure  5. 

Input  Data 

Identical  to  Run  1  except:       N    =    15. 

Terminal  Performance  Data 


z(N)  = 


.0305 

42.5010 

.0833 

2.5000 


Cost R     =    90.117 
Cost Q     =         .101 


J      =    90.278 

The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 
that: 

a)  the  terminal  "miss-distance"  is  larger,  and 

b)  larger  magnitudes  of  acceleration  are  required. 

The  series  of  feedback-gain  matrices  is  identical  to  that  for  Run  1  . 
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Figure  5 

Intercept  Trajectories 
for 
Reduced  Final  Time 
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Run  4  shows  the  effect  of  reducing  the  magnitude  of  the  elements 
of  the  final-state  weighting  matrix,  Q.    The  input  data  and  the  terminal 
performance  data  are  given  below.    The  resulting  trajectories  are  shown 
in  Figure  6 . 

Input  Data 

Identical  to  Run  1  except: 

|   0    0    0 
0    0    0    0 


0    0    |    0 


Q  - 

0   0    0    0 
Terminal  Performance  Data 


z(N)  = 


.1874 

29.9950 

.0937 

2.4977 


Cost  R     =    23.408 
Cost  Q     =         .022 


that: 


J      =    23.430 
The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 

a)  the  terminal  "miss-distance"  is  larger,  and 

b)  the  required  acceleration  magnitudes  are  smaller. 
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b 


§ 


§ 


Figure  6 

Intercept  Trajectories 
for 
Reduced  Final  State  Error  Weighting 
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Run  5  shows  the  effect  of  making  the  final-state  weighting 
matrix,  Q,  an  identity  matrix.    The  input  data  and  the  terminal  perform- 
ance data  are  given  below.    The  resulting  trajectories  are  shown  in 
Figure  7 . 

Input  Data 

Identical  to  Run  1  except: 


Q  = 


10  0  0 
0  10  0 
0  0  10 
0      0      0      1 


Terminal  Performance  Data 


?(N)  = 


.5440 

6.0027 

.0844 

.5004 


Cost R     =    168.20 
Cost  Q     =      36.59 


J      =    204.79 

The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 
that: 

a)  the  terminal  "miss-distance"  is  larger, 

b)  the  terminal  velocity  difference  vector  is  smaller,  and 

c)  the  required  acceleration  for  this  strategy  is  no  longer  always 
parallel  to  the  projected  terminal  "miss-distance"  vector. 

The  use  of  this  particular  Q  matrix  defines  both  position  and 
velocity  as  "states-of-interest"  for  the  performance  index.     Com- 
paring the  terminal  performance  data  and  trajectories  for  this  run  with 
those  for  Run  1  shows  a  resulting  "rendezvous"  instead  of  an 
"intercept"  . 
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Figure  7 


Uncooperative  Rendezvous  Trajectories 
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Run  6  shows  the  effect  of  decreasing  the  magnitude  of  the  elements 
of  the  evasion  control  weighting  matrix,  R    .    The  input  data  and  the 
terminal  performance  data  are  given  below.    The  resulting  trajectories 
are  shown  in  Figure  8. 

Input  Data 

Identical  to  Run  1  except: 

3      01 
0      3 


R     = 
e 


Terminal  Performance  Data 


z(N)  = 


.1125 

30.0010 

.0563 

2.5005 


Cost  R     =    28.111 
Cost Q     =         .016 


that: 


J      =    28.127 
The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 

a)  the  terminal  "miss-distance"  is  larger,  and 

b)  the  required  acceleration  for  the  evader  is  larger. 

The  physical  implication  of  this  reduction  in  the  magnitude  of 

the  elements  of  the  R    matrix  is  to  make  the  evader  more  maneuver- 

e 

able . 
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Figure  8 

Intercept  Trajectories 
for 
Reduced  Evasion  Control  Weighting 
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Run  7  shows  the  effect  of  increasing  the  magnitude  of  the  elements 
of  the  pursuit  control  weighting  matrix,  R    .    The  input  data  and  the 
terminal  performance  data  are  given  below.    The  resulting  trajectories 
are  shown  in  Figure  9. 

Input  Data 

Identical  to  Run  1  except: 


R     = 
P 

2 

0 

0 
2 

al  Performance 

Data 

"-      .2498' 

Cost  R 

=    62.383 

29.9910 

Cost  Q 

.078 

-      .0469 

2.4977 

J 

=    62.461 

z(N)  = 


The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 
that: 

a)  the  terminal  "miss-distance"  is  larger,  and 

b)  the  required  acceleration  for  the  pursuer  is  less. 

The  physical  implication  of  increasing  the  magnitude  of  the 

elements  of  the  R     matrix  is  to  make  the  pursuer  less  maneuverable 
P 
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Figure  9 

Intercept  Trajectories 

for 

Increased  Pursuit  Control  Weighting 
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Run  8  shows  the  effect  of  decreasing  the  sampled-data  period 
without  changing  the  final  time.    The  input  data  and  the  terminal  per- 
formance data  are  given  below.    The  resulting  trajectories  and  the 
time  variation  of  the  (1,1)  elements  of  the  feedback-gain  matrices 
are  shown  in  Figures  10  and  11. 

Input  Data 

Identical  to  Run  1  except: 


T  =  *  ,      N  =  40   ,      R     = 


0 


i 

2 

o   i 


R     = 
e 


91 

c  2 


0 
?  h 

L   g 


Terminal  Performance  Data 


z(N)  = 


.0937 

29.9950 

.0469 

2.4997 


Cost  R     =    23.419 
Cost  Q     =         .016 


J      =    23.430 

The  resulting  trajectories  differ  from  the  standard  of  Run  1  in 
that: 

a)  the  terminal  "miss-distance"  is  slightly  less,  and 

b)  the  acceleration  required  for  both  plants  is  slightly  less. 
The  trajectories  for  this  run  are  almost  identical  to  those  for 

Run  1 ,   showing  only  slight  improvement  in  performance  for  the  smaller 
sample  period.    This  demonstrates  the  ability  to  use  this  method  with 
small  sample  periods  to  approximate  the  solution  for  a  system  with 
continuous  controls  and  time  varying  feedback-gain  matrices  . 
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Figure  10 
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Time  Variation  of  the  (1,1)  Elements 
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Run  9  shows  the  effect  of  reducing  the  magnitude  of  the  elements 
of  the  evasion  control  weighting  matrix,  R    ,  enough  to  violate  the 
necessary  conditions  for  the  existence  of  the  required  saddle  point. 
The  input  data  and  the  terminal  performance  data  are  given  below.     The 
resulting  trajectories  are  shown  in  Figure  12. 

Input  Data 

Identical  to  Run  1  except: 


R     = 
e 


.4        0 
0      .4 


Terminal  Performance  Data 


z(N)  = 


.0500 

30 

.0132 

.0250 

2 

.5066 

Cost  R     =    -12.514 
Cost  Q     =  .003 

J      =    -12.511 


The  test  for  the  necessary  conditions  for  the  saddle  point  failed 
on  the  second  iteration  within  subroutine  OPTCON2  . 

The  effect  of  the  failure  to  satisfy  the  necessary  conditions  for 
the  saddle  point  is  easily  seen  in  the  trajectories  of  Figure  12  .    In 
this  case  the  pursuer  runs  from  the  evader,  and  the  evader  pursues 
the  pursuer.     Further  study  is  indicated  regarding  violations  of  the 
necessary  conditions  for  the  saddle  point. 
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Figure  12 

Intercept  Trajectories  Illustrating 
Violation  of  Necessary  Conditions 


06!  081  G03  004.  065  006 


V5CALE  -  LME+6Q  UNJTS'INCH. 
T-JCflLE  ■   :.0tt>82  ifil  iT/'INCH 

NORTH  US  EAST,  PURSUIT  "EUflSION  TRAJECTORIES 
L1UTOiIS0N,PERMEN~ER,628,0PT,1L.2     RUN  9     l   1  67 


42 


Run  10  is  made  to  determine  the  point  at  which  decreasing  the 

magnitude  of  the  elements  of  the  evasion  control  weighting  matrix, 

R    ,  causes  failure  of  the  test  for  the  required  saddle  point.     No 
e 

trajectories  are  simulated.     The  strategy  was  computed  for  input  data 

identical  to  that  for  Run  1  except  for  the  values  of  the  R      matrix 

e 

which  are  given  below.    The  existence  of  the  saddle  point  was  tested 
for  100  sample  periods,  and  the  results  are  as  follows: 


saddle  point  exists  for  N  £    100; 


2.7 

0 

R     = 
e 

0 

2.7 

"2.6 

0 

R     = 
e 

0 

2.6 

'2.5 

0 

R    = 
e 

0 

2.5 

"0.4 

0 

R    = 

e 

0 

0.4 

saddle  point  exists  for  N    £    100; 


saddle  point  does  not  exist  for  N  ^  3;. 


saddle  point  does  not  exist  for  N    ^2. 


This  run  shows  that  the  saddle  point  exists  for  the  plants  specified 

for  this  series  of  tests,  with  Q  and  R    as  specified,  for  the  elements 

P 

of  the  R     matrix  greater  than  2.5. 
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5.2   Comparisons  of  Optimal  and  Sub-optimal  Strategies 

Run  1 1  shows  the  effect  of  the  evader  abandoning  the  optimal 
strategy  by  not  maneuvering  at  all.  The  input  data  is  identical  to 
that  for  Run  1  .    A  statement  was  added  in  the  program  to  require: 


v  = 


for  trajectory  simulation. 
The  pursuer  followed  the  optimal  strategy  (assuming  the  evader  would 
do  likewise).    The  resulting  trajectories  are  shown  in  Figure  13. 
Terminal  Performance  Data 


z(N)  = 


.0048 

28 

.5051 

.0024 

1 

.7527 

Cost R     =    19.708 
Cost  Q     =         .000 

J      =    19.708 


Comparison  of  the  results  of  this  test  with  those  of  Run  1  shows 
that  (at  least  for  this  case)  the  left-hand  inequality  of  equation  (13) 
does  indeed  hold.    Intuitively,  the  pursuer  does  "better"  if  the 
evader  does  not  use  his  "intelligence".    The  series  of  feedback- 
gain  matrices  is  identical  to.  that  for  Run  1 . 
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Figure  13 
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Run  12  shows  the  effect  of  "improving"  the  pursuer's  knowledge 
of  the  evader.  The  input  data  and  the  terminal  performance  data  are 
given  below.    The  resulting  trajectories  are  shown  in  Figure  14. 

Input  Data 

Identical  to  Run  1  and  Run  11  except: 


B     = 
e 


(In  this  case  the  statement  v  =  0  is  not  required 
to  prevent  the  evader  from  maneuvering.) 


0 

0 

0 

0 

0 

0 

.0 

0 

Terminal  Performance  Data 


z(N)  = 


-  .0750 
30.0041 

-  .0375 
2.5019 


Cost R     =    18.748 
Cost Q     =         .007 

J      =    18.755 


Comparison  of  the  results  of  this  run  with  those  of  Run  11 
shows  that  the  pursuer  can  indeed  do  "better"  if  he  has  (or  uses) 
"better  intelligence".     Note  that  in  this  case  "better"  means  a  lower 
value  of  the  performance  index,  and  not  necessarily  a  smaller 
"miss-distance" . 
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Run  13  shows  the  effect  of  the  pursuer  using  the  strategy  of 
"proportional  navigation"  against  the  same  non-maneuvering  evader. 
The  input  data  is  identical  with  that  for  Run  1  and  Run  11 .    The  evader 
is  again  prevented  from  maneuvering  by  inserting  a  statement  re- 
quiring v  =  0^  for  the  trajectory  simulation  as  in  Run  11 .    Additional 
statements  are  added  to  require  the  pursuer's  speed  to  remain  con- 
stant, and  his  rate  of  turn  to  be  five  times  the  evaders  bearing  rate 
(as  seen  from  the  pursuer).     The  resulting  trajectories  are  shown  in 
Figure  15  . 

Terminal  Performance  Data 


z(N)  = 


-269.53 
6.91 

-  68.31 

-  3.34 


Cost R     =    3.975 
Cost  Q     =    7.73  x  10' 


J      =    7.73  x  10 


The  resulting  trajectories  show  vividly  the  common  pitfall  of 
trying  to  compare  two  strategies  based  on  different  assumptions. 
Here  the  fault  is  in  attempting  to  compare  the  derived  strategy,  which 
allows  acceleration  in  any  direction  and  requires  intercept  at  a 
specified  time,  with  the  strategy  of  proportional  navigation,  which 
assumes  the  pursuer  and  evader  are  restricted  to  constant  speeds  (the 
pursuer  faster  than  the  evader)  and  allows  intercept  at  any  time. 


48 


Figure  15 

Intercept  Trajectories 
for 
Non-maneuvering  Evader 
and 
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5.3  Application  to  the  One-sided  Control  Problem 

Run  21  simulates  the  resulting  trajectory  of  a  single  system,  using 
evader  dynamics  to  specify  the  state  of  the  pursuit  system  desired  at 
the  final  time.    The  state  vector  for  the  system  is  defined: 


x  = 


position 
velocity 
acceleration 
acceleration  rate 


(52) 


The  dynamics  for  the  pursuit  plant  are  those  of  an  arbitrary,  stable, 

fourth-order  system.    The  dynamics  of  the  evasion  plant  maintain 

x     constant.     The  input  data  and  numerical  results  are  given  below. 
— e 

The  resulting  trajectory  is  shown  in  Figures  16  and  17.     The  time 
variation  of  the  (1,1)  element  of  the  feedback-gain  matrix  is  shown  in 
Figure  18. 


A    = 
P 


Input  Data 

0 
0 
0 
-0.15585 


1.0 

0 
0 
-0.69454 


0 

1.0 
0 
-1.56330 


0 

0 

1. 

0 

-1 

75930 

B   = 
P 


A     = 
e 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Q  = 


1000  0 

0  1000 
0  0 

0  0 


0  0 

0  0 

1000  0 

0  1000 


R      =    1 

p 


x    (0)  = 


100 
10 

0 
0 


X       = 


40 

20 

10 

0 


N  =  50 


T  =  0.1 
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Numerical  Results 


z(N)  = 


1.1185 
-3.4193 

2.3391 
-1.4372 


Cost  R     =    12816 
Cost  Q     =      2048 

J      =    14864 


The  results  of  this  run  demonstrate  that  the  strategy  may  be 
successfully  used  to  cause  a  plant  to  approach  a  desired  state  at  a 
given  time,  and  that  the  computer  program  is  capable  of  simulating 
the  results  of  this  application. 

The  complete  input  data,   showing  required  format,  and  the  com- 
puter printout  of  the  numerical  results  of  this  run  are  given  in  Appendix 
B.    The  changes  made  in  the  main  program  to  provide  a  useful  form 
of  graphical  computer  output  for  this  run  are  shown  in  Appendix  C. 
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Figure  16 

Position- Velocity-Acceleration  Vs  .  Time 
for  Single  Plant 
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Figure  17 

Position  Vs.  Velocity  Phase  Trajectory 
for  Single  Plant 
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Figure  18 

Time  Variation  of  the  (1,1)  Elements 
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6.    CONCLUSIONS 

A  class  of  pursuit-evasion  games  has  been  solved,  illustrating 
the  application  of  differential  game  theory  and  dynamic  programming 
to  control  system  design.    A  computer  program  has  been  provided  to 
simulate  the  response  of  linear  systems  to  controls  based  on  the 
resulting  strategy.    A  set  of  necessary  and  sufficient  conditions  for 
the  existence  of  the  solution  has  been  found.    Attempts  to  simplify 
these  conditions  and  to  provide  an  adequate  physical  interpretation 
of  them  have  been  unsuccessful. 

That  this  strategy  successfully  generates  feedback  controls  for 
pursuit  and  evasion  plants,  for  which  the  system  dynamics  are  linear, 
and  linearly  related,  and  for  which  some  form  of  "intercept"  or 
"rendezvous"  is  desired,  has  been  demonstrated.    A  certain  general- 
ity of  application  of  the  strategy  is  inherent  in  the  ability  to  choose 
different  plants  for  the  pursuer  and  evader,  the  ability  to  choose 
different  combinations  of  weighting  matrices  in  the  performance 
index,  and  the  ability  to  handle  arbitrary  initial  conditions.    If  the 
plant  dynamics  and  performance  index  are  stated  beforehand,  the  F    , 

F    ,  0     ,  and  0      matrices  may  be  computed  and  tabulated,  making 
e       p  e 

the  "on-line"  implementation  of  the  strategy  mathematically  simple 
and  rapid. 

The  requirement  for  a  linear  relationship  between  the  dynamics  of 
the  two  plants  is  restrictive.    For  example,  a  linear  representation  of 
the  dynamics  of  one  aircraft  about  its  roll,  pitch,  and  yaw  axes  is 
quite  realistic  [  7  ]  .     However,  the  dynamics  of  two  maneuvering  air- 
craft expressed  about  their  respective  axes  are  not  linearly  related. 
A  limitation  is  also  imposed  on  the  desired  system  response  by  the 
form  of  the  performance  index.     For  example,  an  aircraft  commander 


who  finds  himself  on  a  collision  course  with  an  intercepting  missile 

T 
(z(k)   Qz_(k)    =    0),  may  find  the  application  of  this  strategy,  which 

tells  him  to  conserve  his  fuel  (v(k)    =    F  z(k)    =    0),  very  unsatisfying 


55 


An  obvious  area  for  future  investigation  is  the  extension  of  this 
strategy  to  include  the  broader  class  of  pursuit-evasion  problems 
having  a  performance  index  of  the  quadratic  form: 

J    =    £x   (U  "  x    (t,)]TQ[x    (t  )  -  x    (t  )] 
— p    f       — e    f  — p    f       — e    f 

+  L      CLxn(t)  -  x   (t)]XS[x   (t)  -x   (t)] 
°0  — p  — e  — p  — e 


+  u(t)TR  u(t)  -  v(t)TR  v(t)}dt  (53) 

p-  -  e- 


Simplification  of  the  necessary  and  sufficient  conditions  for 
the  existence  of  a  saddle  point  and  the  physical  interpretation  of 
these  conditions  is  another  area  for  future  investigation.    Ho  in- 
timates that  these  conditions  are  related  to  the  "relative  control- 
lability" of  the  two  plants  [  3  ]  ,  which  suggest  a  method  of  attack 
and  the  possibility  of  gaining  additional  insight  into  the  design  of 
pursuit-evasion  systems  in  general. 
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APPENDIX  A 

DIGITAL  COMPUTER  PROGRAM  FOR  TESTING 
THE  OPTIMAL  PURSUIT-EVASION  STRATEGY 
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APPENDIX  C 

SAMPLE  VARIATIONS 

OF 
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